Skip to content

Accept valid XML documents the token scan wrongly rejected - #175

Merged
jeremy merged 1 commit into
mainfrom
xml-valid-documents
Aug 22, 2026
Merged

Accept valid XML documents the token scan wrongly rejected#175
jeremy merged 1 commit into
mainfrom
xml-valid-documents

Conversation

@jeremy

@jeremy jeremy commented Aug 22, 2026

Copy link
Copy Markdown
Member

Follow-up to #173. The root-element scan held the SAX-parity line too tightly in three places, rejecting documents a namespace-aware SAX parser accepts — safe-direction misses, but real gaps in the advertised parity, confirmed against OpenJDK 17's namespace-aware SAX parser:

Leading zeros in character references. The CharRef grammar has no digit cap, so 	 is legal. Digit runs are now unbounded in the scanner and bounded lexically — at most seven significant decimal digits or six hex, covering every code point through U+10FFFF — before conversion, so an unbounded digit run is never converted to an integer. All zeros denote #x0, still rejected.

PI targets and entity names use the full XML Name grammar. Namespace processing claims the colon in QNames and prefixes only; in a PI target or entity name it is an ordinary name character, so <?p:x?> and a DTD-covered &p:x; are valid. A new XML_NAME grammar (the NCName classes plus colon) is routed through skip_processing_instruction and valid_references?; QName parts and xmlns: prefixes stay NCName, and the reserved-xml-target check is untouched.

Legacy multibyte encodings are transcoded before scanning. Shift_JIS ソ is 83 5C — its trail byte is an ASCII backslash — so the raw byte scanner stopped mid-character inside names, attribute values, and DOCTYPE subsets. Documents declaring any encoding other than UTF-8 are now replace-transcoded to UTF-8 up front; a U+FFFD in the consumed prefix (through the end of the root start-tag) marks input the parser would reject, while replacements past the root — trailing garbage, a character split by MAX_SCAN — are never examined, preserving the pinned "invalid bytes after the root are harmless" boundary. One deviation from the narrow fix: declared US-ASCII also takes the transcode path rather than staying raw, which is behavior-identical (ASCII transcodes 1:1; high bytes become U+FFFD and reject exactly as the old valid_encoding? check did; US-ASCII cannot encode a genuine U+FFFD) and leaves the raw path purely UTF-8. The rare charset that can encode U+FFFD itself, like GB18030, only forfeits a refinement — noted in a comment.

With every scanned document UTF-8, valid_name? no longer threads the document encoding and valid_text? takes a text mode instead.

Probes cover each gap in both directions — the valid documents now refine, and the invalid neighbors (all-zero scalars, oversized references, undeclared &p:x;, bytes invalid in the declared encoding, US-ASCII high bytes) still hold the generic type — plus the boundary pins: invalid bytes stay fatal before the root and harmless after it, including a multibyte character split exactly at the scan limit. Full suite: 664 runs, 1684 assertions, 0 failures.

The root-element scan held the SAX-parity line too tightly in three
places, rejecting documents a namespace-aware SAX parser accepts —
safe-direction misses, but real gaps in the advertised parity:

* Character references admit unlimited leading zeros (the CharRef
  grammar has no digit cap), so &#9; is legal. Digit runs are
  now unbounded in the scanner and bounded lexically — significant
  digits only — before conversion, never by converting an unbounded
  digit run.

* PI targets and entity names use the full XML Name grammar, where the
  colon namespace processing claims in QNames and prefixes is an
  ordinary name character: <?p:x?> and a DTD-covered &p:x; are valid.
  NCName still governs QName parts and xmlns prefixes.

* Legacy multibyte encodings may use ASCII bytes as trail bytes
  (Shift_JIS ソ is 83 5C, ending in an ASCII backslash), derailing the
  raw byte scanner inside names, values and DOCTYPE subsets. Such
  documents are now replace-transcoded to UTF-8 up front; a U+FFFD in
  the consumed prefix marks input the parser would reject, while
  replacements past the root — trailing garbage, a character split by
  the scan limit — stay harmless, preserving the pinned boundary.

With every scanned document UTF-8, valid_name? no longer threads the
document encoding and valid_text? takes a text mode instead.
@jeremy
jeremy merged commit f6f5ce6 into main Aug 22, 2026
20 checks passed
@jeremy
jeremy deleted the xml-valid-documents branch August 22, 2026 08:48
@jeremy jeremy mentioned this pull request Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant